In recent years, three-dimensional computer graphics techniques for generating a three-dimensional image seen from an arbitrary point based on shape data of an object represented by three-dimensional coordinates have been extremely important in fields such as article design or simulation of physical events. Further, techniques for compositing an image generated by the use of the three-dimensional computer graphics techniques, and a photo image have been effective in fields of video production such as television broadcasting, cinema, and the like. A description will now be given of an image generation apparatus and an image composition apparatus which have been employed for such use, according to a prior art.
First, a description will be given of an image generation apparatus for generating an image based on three-dimensional shape data. A general image generation apparatus illustrated herein according to the prior art, handles an object to be displayed as a plane polygon (in many cases "triangle") called "polygon ", and uses shape data described as three-dimensional coordinates.
FIG. 17 is a block diagram showing an image generation apparatus according to a prior art. Turning to FIG. 17, this image generation apparatus comprises a shape data buffer 1701, a pixel generating unit 1702, a Z buffer 1103, a frame buffer 1704, and an image display 1705. The shape data buffer 1701 holds shape data input to the image generation apparatus. As mentioned previously, the shape data is described as the three-dimensional data representing the polygon. The pixel generating unit 1702 sequentially reads shape data for each polygon held in the shape data buffer 1701, divides the read data in display pixel units, and finds a display color of each display pixel as a pixel value (R, G, B). The pixel generating unit 1702 performs an operation according to a formula considering a view point, to find a depth value (Z value) indicating a position on a display screen as depth in the direction of line of sight.
Algorithms for deciding the value of the pixel to be generated, includes "flat shading" in which all the pixels belonging to the same polygon have the same value, and "smooth shading" for calculating a value for each pixel belonging to the same polygon according to a formula considering environmental capabilities such as a light source, and the like. The flat shading is a simple algorithm, arid Therefore reduces processing burden. However, since the flat shading makes it difficult to obtain a display image having smooth and fine shades, the smooth shading is generally used to obtain a realistic and high-quality image. Such processing is described in detail in "Computer Graphics PRINCIPLES AND PRACTICES, Foley, by van dam, Feiner, Hughes, ADDISON-WESLEY PUBLISHING COMPANY (1990)".
The pixel generating value unit 1702 is implemented by software by using a general-purpose processor, although it may be implemented by dedicated hardware. Moreover, the dedicated hardware is commonly implemented by an IC (integrated circuit). In this case, the IC includes a control function for reading data from the frame buffer 1704 which is mentioned later, and a D/A conversion function. The Z buffer 1703 and the frame buffer 1704 in FIG. 17 hold the depth value (Z value) and the pixel value (R, G, B), respectively. The Z buffer 1703 holds Z values of display pixels and the frame buffer 1704 holds pixel values (R, G, B) of the display pixels.
In computer graphics process which handles a display object as a polygon, "hidden-surface" process which prevents an invisible part or an invisible object seen from a certain view point from being displayed, improves reality. The image generation apparatus uses a "hidden-surface" method termed a "Z buffer method". In the process according to the Z buffer method, a Z value of a polygon previously processed is held in the Z buffer, which is compared to a Z value of a polygon to be processed, to decide whether or not the polygon to be processed is positioned in front, thereby deciding whether or not the polygon should be displayed. In the image generation apparatus in FIG. 17, comparison is made between the Z value of the polygon generated by the pixel generating unit 1702 and the Z value of the corresponding pixel of the previously processed polygon held in the Z buffer 1703. When the Z value of the polygon is smaller that that of the pixel, the Z value held in the Z buffer is replaced by the Z value of the polygon, and then the value held in the frame buffer 1704 is replacedby the pixel value of the polygon. A smaller Z value indicates a smaller depth, and therefore the above processing can display the polygon to be seen from a certain view point and can eliminate the polygon which cannot be seen therefrom without being displayed.
For the hidden-surface method, algorithms other than the Z buffer method is possible. For example, there is a conventional "depth sorting method". In the depth sorting method, all the polygons are rearranged in the order of depth, and then pixel values of the polygons are written to a frame buffer (in increasing or decreasing order of depth). An advantage of this method is that a memory as a buffer (Z buffer) for holding depth values is dispensed with. However, processing amount of rearrangement is greatly increased with an increase in the number of polygons, since "order" of the number of polygons n is n.sup.2 or n.times.log n. The Z buffer method is suitable for handling many polygons rather than the depth sorting method. To obtain high-speed by the concurrent execution of such processing, the-Z buffer method is effective, because it can perform individual processing in parallel for each polygon.
In general, the image display 1705 is a CRT (cathode ray tube) monitor, and displays an image generated by the image generation apparatus. In some uses of the image generation apparatus, a monitor or the like as the image display 1705 may be replaced by video editing equipment.
When the shape data, i.e., the three-dimensional coordinates are input to the image generation apparatus so constructed, the shape data is first held in the shape data buffer 1701. The pixel generating unit 1702 reads shape data for each polygon from the shape data buffer 1701, and generates the pixel value and the depth value as mentioned previously. The pixel generating unit 1702 makes a comparison between the generated Z value and the Z value of the corresponding pixel of the previously processed polygon, and then rewrites a content of the Z buffer 1703 when the generated value is smaller.
After rewriting, the generated pixel value (R, G, B) is output to the frame buffer 1704, followed by rewriting a content thereof.
The pixel values held in the frame buffer 1704 are read in the order of the corresponding pixels, and displayed on the image display 1705. Alternatively, the pixel values may be first subjected to D/A conversion process and then output to the image display 1705.
The prior art image generation apparatus is thus capable of generating an image which can be displayed in a three-dimensional manner, by the use of the Z buffer which performs hidden-surface method according to the Z buffer method.
Disadvantages of the Z buffer method are that "anti aliasing" process is difficult and representation of transparent objects is difficult because it determines the algorithm from the depth value. Accordingly, there have been problems associated with the prior art image generation apparatus which uses the Z buffer method, that is,"1. Problems With Anti Aliasing Process" and "Problems with Handling Translucent Objects", which will be described below.
"1. Problems With Anti Aliasing Process"
In a case where the prior art image generation apparatus so constructed generates a computer graphics image, all the display pixels displays a specific polygon, that is, they belong to the specific polygon. At a boundary between polygons, a pixel belonging to a polygon has a value significantly different from a value of a pixel belonging to another polygon which is adjacent to the former pixel. For instance, in a case where an image is displayed in a resolution of a normal TV screen, a size of a pixel is recognizable by a viewer who sees the image. Therefore, in the case where the values of pixels significantly differ from each other, luminance greatly changes at the boundary, which becomes conspicuous. As a consequence, "jaggy" occurs at boundaries of oblique lines or curved lines, which makes it impossible to realize smooth display, causing degraded image quality.
FIGS. 18(a) and 18(b) show such phenomenon and anti aliasing process as measures against it. FIG. 18(a) shows that the "jaggy" damages the display. At a boundary between an upper white polygon and a lower black polygon, there are steps. To avoid this, as shown in FIG. 18(b), colors of these upper and lower polygons at the boundary may be mixed. The mixing ratio is made directly proportional to areas of each polygon in each pixel. This processing is the anti aliasing process for improving image quality.
FIG. 19 shows such anti aliasing process. To find the mixing ratio for use by the anti aliasing process, a display pixel is divided into "sub-pixels" as units, and a polygon is detected for each sub-pixel. In the example shown in FIG. 19, a pixel is divided into 16 sub-pixels (4.times.4 sub-pixels). In sub-pixels in a region represented by oblique lines and sub-pixels in another region, different polygons are present. The mixing ratio is the number of sub-pixels/ a pixel. For example, for a pixel present in a lower-rightmost position (16 sub-pix ls), the mixing ratio of the polygon in the region represented by oblique lines (9 sub-pixels )is 9/16.
In this method, the more sub-pixels a pixel is divided into, the closer to the area ratio of the polygon the mixing ratio becomes. In the hidden-surface method by the use of the Z buffer, there is no problem with algorithm as long as anti aliasing process is carried out by using sub-pixel division. However, such anti aliasing process requires Z buffers and frame buffers for all the sub-pixels on a display screen, and correspondingly requires a large memory capacity.
For instance, suppose that the Z value is 24 bits, each of the R value, the G value, and the B value is 8 bits, and the display screen is composed of (1024.times.512) pixels. In this case, the Z buffer and the frame buffer must respectively have a memory capacity of 1.5 Mbytes, which amounts to 3 Mbytes in total. If a pixel is divided into 16 sub-pixels as shown in FIG. 19 under the same condition, 48 M bytes (3 M.times.16) is required.
The increase in the memory capacity leads to cost-up, and hence in the prior art image generation apparatus, generation of smoothly displayed images is not compatible with a reduction of cost.
"Problems with Handling Translucent Objects"
There are many methods which represent translucent objects such as a color glass by using computer graphics image generation. For the case of handling this kind of object, a parameter called "degree of transparency" is set in shape data, and the degree of transparency is given to each polygon, for representation. As the degrees of transparency, numeric values 0-255 are used. In this case, a value "255" indicates that the corresponding object is opaque, a value "0" indicates that the corresponding object is completely transparent (invisible), and an intermediate value indicate that a degree of transparency varies according to the value.
As concerns a pixel on a display screen, when a polygon having the smallest Z value, of plural polygons including the pixel, is translucent, that is, the degree of transparency is not 255, a display color of the pixel is obtained by mixing a display color of the polygon and a display color of a polygon having the second smallest value according to the degree of transparency. If the polygon having the second smallest Z value is translucent, a display color of a polygon having the third smallest Z value must be used.
However, in the hidden-surface process by the use of the Z buffer, coordinates closest to a view point, i.e., coordinates having small depth, are selected, and therefore only the display color of the polygon having the smallest Z value is held in the frame buffer. Mixing process for plural polygons becomes impossible with a simple frame buffer and a Z buffer in FIG. 17.
Use of plural frame buffers and Z buffers can realize hidden-surface process by the Z buffer method, and mixing process which represents a translucent object, for the case of overlapped polygons having degrees of transparency.
FIG. 20 shows process which uses plural buffers. In the example in FIG. 20, 4 frame buffers, and 4 Z buffers are shown. Z buffers 2011, 2012, 2013, and 2014, hold the smallest Z value, the second smallest Z value, the third smallest Z value, and the fourth smallest value, respectively. Frame buffers 2021, 2022, 2023, and 2024 hold values (R, G, B) of the corresponding Z buffers 2011-2014. In addition, transparency-degree buffers 2C31, 2032, 2033, and 2034 for holding degrees of transparency for respective pixels are used to perform the mixing process for each pixel.
This construction in FIG. 2C is capable of performing process considering the degrees of transparency, which increases a memory capacity required in proportion to the number of the frame buffers and the Z buffers.
Moreover, in order to realize anti aliasing process and representation of the translucent object simultaneously, a necessary memory capacity becomes a product of increases in these capacities, causing added cost-up. In the example shown, memory of 192 Mbytes, i.e., (a memory capacity of display pixels 3 M bytes).times.64 (=16 times.times.4 times) is required.
As should be appreciated, the problem with the prior art is that the preferable image display is not compatible with a reduction of cost.
Subsequently, a description will be given of a prior art image composition apparatus for compositing the image generated by the three-dimensional computer graphics and the photo image.
FIG. 21 shows a general image composition apparatus according to the prior art Turning to FIG. 21, the image composition apparatus comprises an image generating unit 2101, a depth composition unit 2102, a pixel mixing unit 2103, a composition key processing unit 2104, an image composition unit 2105, and an image display 2106. The image generating unit 2101, the depth composition unit 2102, and the pixel mixing unit 2103, constitute a CG generating unit 2110.
The CC generating unit 2110 receives shape data described as three-dimensional coordinates representing a polygon as an input, and generates a CG image by a computer graphics process. Suppose that the CG generating unit 2110 includes plural buffers to perform anti aliasing process and translucency process, as already described by the use of the prior art image generation apparatus.
In this example, the CC generating unit 2110 comprises the image generating unit 2101, the depth composition unit 2102, and the pixel mixing unit 2103, for performing anti aliasing process and the like. This construction requires a large memory capacity as described above. As in the case of the prior art image generation apparatus in FIG. 17, the apparatus may comprise the shape data buffer, the pixel generating unit, the Z buffer, and the frame buffer, and the anti aliasing process may dispensed with.
The image generating unit 2101 sequentially processes input shape data, to generate a pixel value (R, G, B) and a depth value (Z value) for each pixel. In anti-aliasing process, a pixel is divided into sub-pixels, and a mixing ratio is calculated. In translucency process, the degree of "transparency" is used.
The depth composition unit 2102 performs hidden-surface process according to the Z buffer method. For the anti aliasing process and the translucency process, there are provided plural Z buffers and frame buffers, and plural mixing ratio buffers and transparency-degree buffers as required, to perform the processing in FIG. 20. To be specific, 4 buffers are provided for translucency process as in the case shown in FIG. 2a. Besides, the transparency-degree buffers may be replaced by the mixing ratio buffers, to perform anti aliasing process, thereby performing the processing in FIG. 20.
The pixel mixing unit 2103 performs a mixing process by the use of the pixel value (R, G, B), according to the degree of transparency or the mixing ratio, to generate a value of a pixel to-be-displayed, which is held in the frame buffer. As the Z value of the generated image, the unit 2103 fetches the smallest Z value from the Z buffer, and outputs it to the image composition unit 2105.
The CG image so processed by the CG generating unit 2110 and the photo image are composited by a chromakey method. This method employs difference between hues to generate a chromakey signal, i.e., a sampled signal, and performs composition by fitting another frame into the portion. For instance, an image of a specific object (human being or a car) is taken with a specific color named "chromakey blue" for its background, a portion of the "chromakey blue" is sampled from the resulting image by a prescribed process, and the remaining specific object is overlapped with the background image.
The composition key processing unit 2104 receives the photo image as an input and performs prescribed processing, to generate a chromakey signal and photo image information. The photo image information comprises pixel values (R, G, B) of the photo image. The chromakey signal and the photo image are input to the image composition unit 2105. The depth value (Z value) of the photo image is also input to the image composition unit 2105. The Z value specifies the overlapping relationship of the photo image and the CG image.
The image composition unit 2105 makes a comparison between the Z value of the CG image from the pixel mixing unit 2103 and the Z value of the photo image. When the Z value of the photo image is smaller than that of the CG image, the pixel value held in the frame buffer is replaced by the pixel value included in the photo image, to be held therein. Thereby, the photo image is overlapped with the CG image, to be displayed, when the depth value of the photo image is smaller, that is, the photo image is positioned in front in relation to the CG image. Thereafter, the pixel value held in the frame buffer is read therefrom and is subjected to D/A conversion process, to be displayed on the image display 2106.
Thus, in the prior art image composition apparatus, the CG generating unit 2110 performs anti aliasing process to the CG image to make its "jaggy" inconspicuous, or performs translucency process thereto. However, the following problems occur.
i) Since the anti aliasing process cannot be performed to the photo image and the composite image, the image quality is sometimes degraded. For instance, when the photo image is positioned behind the CG image, its boundary portion looks blurred, resulting in degraded image quality.
ii) Flexible composition process cannot be performed. Assume that the translucency process is performed in the CG generation process, and a CG image in which a translucent object is positioned before an opaque object is obtained. When this CG image and a photo image are composited, although the photo image can be disposed before or after the CG image, it cannot be disposed between the translucent object and the opaque object. In other words, while flexible processing which can use overlapping relationships of the number of levels equal to the number of buffers (4 levels in the example shown in FIG. 20) is possible in CG generation process, such flexible processing is impossible in the process for compositing the CG image and the photo image.
It the image composition apparatus comprises plural CG generating units 2110 in FIG. 21 for such flexible composition process, then the composition process can increase the number of levels equal to the number of CG generating units. An example of such composition process is that a CG image output from a CG generating unit is a background imago of a photo image, and a CG image output from another CG generating unit is disposed before the photo image. As described above, the CG generating unit 2120 corresponds to a main part of the prior art image generation apparatus, and therefore plural generating units significantly increases a cost. In particular, plural buffers must be provided for the anti-aliasing process or the translucency process, which requires a large memory capacity, and hence this is not realistic.
As should be appreciated from the foregoing description, in the prior art image generation apparatus, if the anti aliasing process or the translucency process is performed to obtain improved image quality, a required buffer capacity is increased, and correspondingly its cost is increased.
In addition, the anti aliasing process or the translucency process is not applied to the process for compositing the CG image and the photo image. As a consequence, an image quality is not improved and image processing is not performed flexibly.